Papers with visual grounding
Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representations (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study of the effect of visual grounding on language representations has given a new life to the debate around extractability and quality of semantic information in representations trained solely on textual input. |
| Approach: | They compare word embeddings from vision-and-language models to text-only models . they identify meaning properties and relations that characterize words whose embeddements are most affected by visual grounding . |
| Outcome: | The proposed model differs from text-only models on semantic representations of language . the study is the first large-scale study of the effect of visual grounding on language representations . |
Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers (2021.acl-srw)
Copied to clipboard
| Challenge: | a growing body of research has been focused on what attention heads learn during the pre-training of visual grounded language models. |
| Approach: | They propose to use visual grounding to supervise attention directly to learn visual ground. |
| Outcome: | The proposed method improves the performance of a state-of-the-art visual grounded language model on vision-and-language tasks. |
Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) are trained on orders of magnitude more language data than human language learners receive, but without supervision from other sensory modalities that play a crucial role in human learning. |
| Approach: | They propose a grounded language learning procedure that leverages visual supervision to improve textual representations. |
| Outcome: | The proposed procedure outperforms standard language-only models in terms of learning efficiency in small and developmentally plausible data regimes and improves perplexity by around 5% on multiple language modeling tasks compared to other models trained on the same amount of text data. |
MGPO: Thinking with Images via Multi-Turn Grounding-Based Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art large multimodal models face challenges when processing high-resolution images, as these inputs are converted into enormous visual tokens, many of which are irrelevant to the downstream task. |
| Approach: | They propose a multi-turn grounding-based policy optimization framework that enables LMMs to iteratively focus on key visual regions by automatically cropping sub-images based on model-predicted grounding coordinates within a multiple-turn conversation framework. |
| Outcome: | The proposed framework improves on Qwen2.5-VL-7B with 21K samples and surpasses OpenAI’s o1 and GPT-4o models on the out-of-distribution (OOD) V* Bench. |
GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution (2022.coling-1)
Copied to clipboard
Danfeng Guo, Arpit Gupta, Sanchit Agarwal, Jiun-Yu Kao, Shuyang Gao, Arijit Biswas, Chien-Wei Lin, Tagyoung Chung, Mohit Bansal
| Challenge: | Multimodal coreference resolution (MCR) is a crucial capability for building next-generation conversational agents. |
| Approach: | They propose a multimodal coreference resolution model that resolves coreferences made in multi-turn dialogues with scene images. |
| Outcome: | The proposed model resolves coreferences made in multi-turn dialogues with scene images. |
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought (2026.findings-eacl)
Copied to clipboard
| Challenge: | Lack of perceptual grounding limits vision-language models' ability to interpret visual data . prior work on visualized data understanding focused on adapting VLMs to instruction tuning and chain-of-thought supervision . |
| Approach: | They propose a framework that enhances visual reasoning through human-like interpretation grounding. |
| Outcome: | The proposed framework improves on ChartQA and ChartQAPro benchmarks by +11.2%. |
Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on implicit exploration of multimodal coreference but neglect the importance of locating the objects explicitly in the visual content, which is associated with textual entities. |
| Approach: | They propose a multimodal incremental transformer with visual grounding which aims to explicitly locate related objects in the image guided by textual entities. |
| Outcome: | The proposed model achieves comparable performance on the VisDial v0.9 and v1.0 datasets. |
Referring to Screen Texts with Voice Assistants (2023.acl-industry)
Copied to clipboard
Shruti Bhargava, Anand Dhoot, Ing-marie Jonsson, Hoang Long Nguyen, Alkesh Patel, Hong Yu, Vincent Renkens
| Challenge: | a new approach to voice assistants is limited in their ability to understand context of the user. |
| Approach: | They propose a general purpose model that allows users to refer to phone numbers, addresses, email addresses, urls, and dates on their phone screens. |
| Outcome: | The proposed model is lightweight, offering flexibility, better interpretability and efficient run time memory. |
FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts (2024.findings-acl)
Copied to clipboard
Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, Dan Roth
| Challenge: | Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. |
| Approach: | They propose to use flowcharts as visual contexts to assess the capabilities of visual question-answering multimodal language models in reasoning. |
| Outcome: | The proposed benchmarks evaluate models' ability to follow visual information without pre-existing knowledge on a suite of open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. |
VividMed: Vision Language Model with Versatile Visual Grounding for Medicine (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) have demonstrated promise in generating visually grounded responses, but their application in the medical domain is hindered by unique challenges. |
| Approach: | They propose a vision language model with versatile visual grounding for medicine that generates semantic segmentation masks and instance-level bounding boxes. |
| Outcome: | The proposed model can generate semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. |
SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling (2024.eacl-long)
Copied to clipboard
| Challenge: | Visual storytelling aims to automatically generate a coherent story based on a given image sequence. |
| Approach: | They propose a framework that represents the image sequence as a graph with objects and relations that includes human action motivation and its social interaction commonsense knowledge. |
| Outcome: | The proposed framework produces stories superior across multiple metrics in terms of visual grounding, coherence, diversity, and humanness, per both automatic and human evaluations. |
Visually Grounded Neural Syntax Acquisition (P19-1)
Copied to clipboard
| Challenge: | a visually grounded neural syntax learner is an approach for learning syntactic representations without any supervision. |
| Approach: | They propose a visually grounded neural syntax learner that acquires syntax by looking at images and reading captions. |
| Outcome: | The proposed model outperforms unsupervised approaches on the MSCOCO data set . it is more stable with choice of initialization and amount of training data, the authors show . |
GROOViST: A Metric for Grounding Objects in Visual Storytelling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Several evaluation metrics for visual storytelling do not consider images at all . authors propose a novel evaluation tool that accounts for cross-modal dependencies and temporal misalignments . |
| Approach: | They propose a visual storytelling evaluation tool that evaluates visual grounding . they use cross-modal dependencies, temporal misalignments and human intuitions . |
| Outcome: | The proposed evaluation tool accounts for cross-modal dependencies, temporal misalignments and human intuitions on visual grounding. |
Measuring Faithful and Plausible Visual Grounding in VQA (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lack of visual grounding (VG) in VQA systems can manifest in over-reliance on irrelevant image parts or a disregard for the visual modality entirely. |
| Approach: | They propose a new metric that measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
| Outcome: | The proposed metric measures if a model identifies question-relevant objects in the scene and relies on the information contained in the relevant objects when producing its answer. |
RoViST: Learning Robust Metrics for Visual Storytelling (2022.findings-naacl)
Copied to clipboard
| Challenge: | Visual storytelling is the task of generating a story paragraph that describes a given image sequence. |
| Approach: | They propose 3 evaluation metrics sets that analyze which aspects we would look for in a good story . they compare their correlation with human judgement scores on a sample of machine stories . |
| Outcome: | The proposed evaluation metrics outperform other metrics on human correlation on a sample of machine stories from state-of-the-art models. |
Dual Attention Networks for Visual Reference Resolution in Visual Dialog (D19-1)
Copied to clipboard
| Challenge: | Visual dialog (VisDial) requires a dialog agent to answer a series of questions grounded in an image. |
| Approach: | They propose dual attention networks (DAN) for visual reference resolution in VisDial. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model on VisDial datasets. |
BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models (2026.eacl-long)
Copied to clipboard
Bryan Chen Zhengyu Tan, Weihua Zheng, Zhengyuan Liu, Nancy F. Chen, Hwaran Lee, Kenny Tsu Wei Choo, Roy Ka-Wei Lee
| Challenge: | Existing evaluations assess static recall or isolated visual grounding, leaving unanswered whether VLMs possess robust and transferable cultural understanding. |
| Approach: | They propose a multimodal, multicultural benchmark to evaluate the robustness of everyday cultural knowledge in vision-language models across linguistic rephrasings and visual modalities. |
| Outcome: | ‘BLEnD-Vis‘ constructs 313 culturally grounded question templates spanning 16 regions and generates three aligned multiple-choice formats. |
Simple-VGC: Enhancing Visual Grounding in Multimodal Reasoning via Adaptive Tool Composition (2026.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal large language models suffer from systematic failures in basic visual understanding. |
| Approach: | They propose a tool-augmented reasoning framework with three targeted compensation strategies to address these problems. |
| Outcome: | The proposed framework improves visual grounding by re-injecting the original image to mitigate visual forgetting, the authors show . the proposed framework also improves the accuracy of the visual inputs, the researchers show - and the results are promising . |
SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency (2021.naacl-main)
Copied to clipboard
| Challenge: | Current visual question answering models are inconsistent in their understanding of the world . they answer difficult reasoning questions correctly but get associated sub-questions wrong . |
| Approach: | They propose a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image and a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT). |
| Outcome: | The proposed approach improves model consistency by up to 6.5% points over existing approaches while improving visual grounding and robustness to rephrasings of questions. |
VIMI: Grounding Video Generation through Multi-modal Instruction (2024.emnlp-main)
Copied to clipboard
Yuwei Fang, Willi Menapace, Aliaksandr Siarohin, Tsai-Shien Chen, Kuan-Chieh Wang, Ivan Skorokhodov, Graham Neubig, Sergey Tulyakov
| Challenge: | Existing text-to-video diffusion models rely on text-only encoders for their pretraining, restricting their versatility and application in multimodal integration. |
| Approach: | They propose a multimodal conditional video generation framework for pretraining on augmented text prompts and then utilize a two-stage training strategy to enable diverse video generation tasks within a model. |
| Outcome: | The proposed model can synthesize consistent and temporally coherent videos with large motion while retaining the semantic control. |
Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding (2026.acl-long)
Copied to clipboard
| Challenge: | Large Vision–Language Models (LVLMs) integrate visual perception with language understanding, but how vision information contributes to the model’s decoding process remains under-explored. |
| Approach: | They propose a simple training-free decoding method that guides text generation in Large Vision–Language Models by Referencing Vision Tokens. |
| Outcome: | The proposed method leverages the semantic information embedded within vision tokens by projecting it into the text token distribution. |
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat (N19-1)
Copied to clipboard
Ravi Shekhar, Aashish Venkatesh, Tim Baumgärtner, Elia Bruni, Barbara Plank, Raffaella Bernardi, Raquel Fernández
| Challenge: | Existing systems that address the abilities that need to be put to work during conversations are lacking in terms of visual grounding. |
| Approach: | They propose a visually-grounded dialogue state encoder which integrates visual grounding with dialogue system components. |
| Outcome: | The proposed system improves the GuessWhat?! game by combining guessing and asking questions with multi-task learning. |
HisDoc-OCR: Restoring Visual Grounding in MLLMs for Chinese Historical Document OCR (2026.findings-acl)
Copied to clipboard
| Challenge: | Despite multimodal large language models' strong performance on modern document OCR, their application to historical Chinese texts suffers from severe hallucinations, character fabrication, uncontrolled repetition, and semantic drift. |
| Approach: | They propose a multimodal large language model which restores visual grounding through three synergistic strategies: Layout Injection, First-Occurrence Boost, Self-Distilled Attention Focusing and HisDoc-OCR. |
| Outcome: | The proposed model outperforms general-purpose and OCR-specific models on Chinese historical documents. |
GOBench: Stage-Wise Diagnostics and the Visual Paradox in Multimodal Graph Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks fail to represent multimodal problem specifications, score outcomes only and cannot localize where failures occur along the modeling pipeline. |
| Approach: | They propose a Graph Optimization benchmark that aligns multiple modalities with solver-derived oracles and a diagnostic protocol that evaluates intermediate artifacts as well as end results. |
| Outcome: | Graph Optimization benchmark (GOBench) evaluates intermediate artifacts as well as end results . vision reliably increases inference cost, while reliability impact is regime-dependent . current benchmarks fail to represent multimodal problem specifications, fail to localize failures . |
MAKAR: a Multi-Agent framework based Knowledge-Augmented Reasoning for Grounded Multimodal Named Entity Recognition (2025.emnlp-main)
Copied to clipboard
Xinkui Lin, Yuhui Zhang, Yongxiu Xu, Kun Huang, Hongzhang Mu, Yubin Wang, Gaopeng Gou, Li Qian, Li Peng, Wei Liu, Jian Luan, Hongbo Xu
| Challenge: | Existing methods for GMNER fail to address semantic ambiguity caused by polysemy and long-tail distribution of datasets. |
| Approach: | They propose a framework for Grounded Multimodal Named Entity Recognition that leverages a Multimodal Large Language Model to address semantic ambiguity. |
| Outcome: | Extensive experiments show that the proposed framework outperforms existing methods on two benchmark datasets. |
Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Current sycophancy research has largely overlooked its specific manifestations in the video-language domain. |
| Approach: | They propose a video-LLM sycophancy benchmarking and evaluation to evaluate scophancies in video-LLMs. |
| Outcome: | The proposed benchmark evaluates sycophantic behavior in state-of-the-art Video-LLMs across diverse question formats, prompt biases, and visual reasoning tasks. |
From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown emerging capabilities through large-scale training that have made them gain popularity in recent years. |
| Approach: | They propose to perform retrieval across universals and cultural visual grounding tasks to assess cultural diversity across universal and culture-specific local concepts. |
| Outcome: | The proposed benchmarks show that the models perform significantly across cultures, underscoring the need for enhancing multicultural understanding in vision-language models. |
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do (2026.acl-long)
Copied to clipboard
| Challenge: | Existing open-source models often yield only marginal overall improvements, possibly due to an overemphasis on mathematical reasoning at the expense of broader capabilities. |
| Approach: | They evaluate 12 multimodal tasks using 14 non-reasoning models and 8 reasoning models. |
| Outcome: | The proposed method is effective in multimodal reasoning tasks, the authors show . they show that it lacks the ability to maintain deep visual introspection throughout the reasoning process. |
OCID-Ref: A 3D Robotic Dataset With Embodied Language For Clutter Scene Grounding (2021.naacl-main)
Copied to clipboard
| Challenge: | Visual grounding (VG) is a crucial task in natural language processing, computer vision, and robotics. |
| Approach: | They propose a visual grounding task with referring expressions of occluded objects in a OCID-Ref dataset with 2,300 scenes and a point cloud input. |
| Outcome: | The proposed dataset shows that it can handle 2D and 3D signals but referring to occluded objects remains challenging for the modern visual grounding systems. |
Countering Language Drift via Visual Grounding (D19-1)
Copied to clipboard
| Challenge: | Emergent multi-agent communication protocols are different from natural language . a long-standing goal of artificial intelligence research is to develop agents that can cooperate with other agents . |
| Approach: | They propose to use syntactic and semantic constraints to improve communication . they propose to combine these constraints with auxiliary training constraints to reduce language drift . |
| Outcome: | a new study shows that pre-trained agents retain English syntax while learning to convey intended meaning . the proposed training constraints can be used to mitigate language drift . |
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022.emnlp-main)
Copied to clipboard
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si
| Challenge: | Existing pre-trained vision-language models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment. |
| Approach: | They propose a vision-language foundation model with cross-modal skip-connections that can be pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives. |
| Outcome: | The proposed model achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering. |
Benchmarking Diverse-Modal Entity Linking with Generative Models (2023.findings-acl)
Copied to clipboard
Sijia Wang, Alexander Hanbo Li, Henghui Zhu, Sheng Zhang, Pramuditha Perera, Chung-Wei Hang, Jie Ma, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Bing Xiang, Patrick Ng
| Challenge: | Existing models for diverse-mode entity linking (EL) work well on per modality configurations, but it is more challenging to design a unified model for diverse modality. |
| Approach: | They propose a generative diverse-modal model that integrates text, image and table . they propose combining a multimodal encoder-decoder paradigm with a fine-tuning GDMM . |
| Outcome: | The proposed model outperforms state-of-the-art models by 8.51 F1 on average for diverse-modal EL. |
Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods address this issue by introducing an auxiliary task such as visual grounding, cycle consistency, or debiasing. |
| Approach: | They propose a data augmentation pipeline to turn “known” knowledge into training examples for VQA. |
| Outcome: | The proposed model can handle multi-modal information and is based on human-annotated examples. |
Visual Grounding Annotation of Recipe Flow Graph (2020.lrec-1)
Copied to clipboard
Taichi Nishimura, Suzushi Tomori, Hayato Hashimoto, Atsushi Hashimoto, Yoko Yamakata, Jun Harashima, Yoshitaka Ushiku, Shinsuke Mori
| Challenge: | Existing studies have ground visual observations with procedural texts with graphs to understand which objects are aligned with textual descriptions. |
| Approach: | They propose to provide visual grounding annotations to recipe flow graphs by adding bounding boxes to image sequences of recipes and annotating two types of event attributes with each bounding box. |
| Outcome: | The proposed dataset gives visual grounding with workflow’s contextual information between procedural text and visual observation in an indirect manner. |
Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Visual storytelling is a task of generating a story for a sequence of several temporally-ordered images or video frames. |
| Approach: | They propose a method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness. |
| Outcome: | The proposed method improves on the foundation model LLaVA but only slightly compared to TAPM, a 50-times smaller visual storytelling model. |
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has adapted vision-and-language models to generative tasks like image captioning. |
| Approach: | They propose an extension to LXMERT with training refinements to generate images from text. |
| Outcome: | The proposed model can generate images from pieces of text while still being comparable to existing models. |
RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought (2025.acl-long)
Copied to clipboard
Yi Lu, Jiawang Cao, Yongliang Wu, Bozheng Li, Licheng Tang, Yangguang Ji, Chong Wu, Jay Wu, Wenbo Zhu
| Challenge: | Recent advances in multi-modal learning have enhanced MLLMs' ability to reason about visual content. |
| Approach: | They propose a framework that unifies multi-step multimodal reasoning with grounded visual understanding. |
| Outcome: | The proposed framework surpasses state-of-the-art methods by +6.5 gIoU and +9.2 cIou on ReasonSeg and achieves 49.7 mAP on SegInW under zero-shot settings. |
A negative case analysis of visual grounding methods for VQA (2020.acl-main)
Copied to clipboard
| Challenge: | Existing Visual Question Answering (VQA) methods exploit dataset biases and spurious statistical correlations instead of producing correct answers for the right reasons. |
| Approach: | They propose to incorporate visual cues to better ground VQA models . they also propose a regularization effect which prevents over-fitting to linguistic priors . |
| Outcome: | The proposed method outperforms existing methods on the Visual Question Answering (VQA) dataset. |
Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing Vid-LLMs lack robust mechanisms for maintaining grounded spatiotemporal beliefs under conversational feedback. |
| Approach: | They propose a negation-based gaslighting evaluation framework and introduce a benchmark to investigate spatiotemporal sycophancy. |
| Outcome: | The proposed framework evaluates state-of-the-art Vid-LLMs across video understanding tasks. |
Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing training-free methods are vulnerable to the attention sink phenomenon . Existing methods include contrastive decoding and auxiliary expert models . |
| Approach: | They propose a training-free attention intervention that constructs a PAD map to identify semantically core visual regions and applies per-head Median Absolute Deviation Scaling to adaptively control the intervention strength. |
| Outcome: | The proposed intervention improves visual grounding and reduces hallucinations on multiple LVLMs and benchmarks. |
VADE: Visual Attention Guided Hallucination Detection and Elimination (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) are prone to hallucinations, generating outputs that lack grounding in the actual visual data. |
| Approach: | They propose a sequence modelling approach to learn complex sequential patterns from transformer attention maps. |
| Outcome: | The proposed approach achieves an average PR-AUC of 80% in hallucination detection on M-HalDetect and an 5% improvement in hallucinosis mitigation on MSCOCO. |
SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes (2026.acl-long)
Copied to clipboard
Chuhan Wang, Xintong Li, Jennifer Yuntong Zhang, Junda Wu, Chengkai Huang, Lina Yao, Julian McAuley, Jingbo Shang
| Challenge: | Existing preference-based approaches fail to address this challenge by exploiting language priors to bypass visual grounding. |
| Approach: | They propose a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. |
| Outcome: | The proposed framework improves answer accuracy and reasoning faithfulness across seven visual reasoning benchmarks. |
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling (2024.emnlp-main)
Copied to clipboard
| Challenge: | a task-agnostic visual encoding yields minimal performance gains on grounding, but Transformers outperform Mamba at in-context multimodal retrieval. |
| Approach: | They propose to replace Transformers in Visual Language Models with Mamba, a structured state space model that demonstrates promising performance in sequence modeling. |
| Outcome: | The proposed model outperforms Transformers-based models in captioning, question answering, and reading comprehension. |
The Revolution of Multimodal Large Language Models: A Survey (2024.findings-acl)
Copied to clipboard
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara
| Challenge: | Recent advances in large language models have led to the development of multimodal large language model. |
| Approach: | They present a review of recent visual-based Large Language Models and analyze their architectures and alignment strategies. |
| Outcome: | The proposed models can integrate visual and textual modalities while providing a dialogue-based interface and instruction-following capabilities. |
Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite significant progress in multimodal language models, it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. |
| Approach: | They propose to assess vision-language models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
| Outcome: | The proposed benchmark assesses the models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing input-centric solutions fail to reverse this intrinsic mechanism of information loss. |
| Approach: | They propose a Variational Information Flow framework that leverages a probabilistic perspective to model visual saliency relevant to the question-answer pair as a latent distribution. |
| Outcome: | The proposed framework improves general VQA, fine-grained perception and visual grounding. |
Mitigating Hallucinations in VLMs: Enhancing Visual Attention via Head-Wise Perturbation (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision–Language Models (VLMs) have demonstrated strong capabilities in tasks that require joint understanding of text and images. |
| Approach: | They propose a strategy that incorporates head-wise attention perturbation via continuous multiplicative noise coupled with a visual-guided loss focused on vision-sensitive text tokens to promote a more balanced attention distribution. |
| Outcome: | The proposed approach outperforms baseline models on three benchmarks and consistently outperformed the baseline model. |
PAR: Training-Free Positional Perturbation and Attention Recycling for Faithful OCR (2026.acl-long)
Copied to clipboard
| Challenge: | In high-precision tasks, vision language models suffer from Linguistic Priors Hallucination . |
| Approach: | They propose a training-free, inference-time intervention framework to mitigate this by integrating visual encoders with Large Language Model decoders. |
| Outcome: | The proposed framework reduces hallucination rates by 12% in long-context scenarios while maintaining robust generalization on standard benchmarks. |
E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches decouple textual entity recognition and visual grounding, leading to error accumulation and suboptimal joint optimization. |
| Approach: | They propose a fully end-to-end generative framework that unifies recognition, semantic typing, visual grounding and implicit knowledge reasoning within a single multimodal large language model. |
| Outcome: | The proposed framework achieves highly competitive performance compared with state-of-the-art methods. |
Look and Think: Efficient Multimodal Reasoning via Modality-Decoupled Compression (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models have strong performance on visual question answering benchmarks . however, their inference efficiency is severely constrained by the rapidly growing context . |
| Approach: | They propose a modality-decoupled compression method that enables efficient multimodal inference . they propose to evict visual tokens whenever visual grounding is unnecessary . |
| Outcome: | The proposed method reduces the average context length by up to 57% while maintaining comparable performance to the standard MLLM baseline. |
Can VLMs Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models integrate textual and visual information, enabling them to process visual inputs and generate predictions. |
| Approach: | They review work on modality collapse analysis to provide insights into the reason for this unintended behavior and review probing studies for fine-grained vision-language understanding. |
| Outcome: | The proposed models can achieve competitive performance in vision-language tasks despite relying heavily on textual information and ignoring visual information. |
GeoChain: Multimodal Chain-of-Thought for Geographic Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Leveraging 1.46 million Mapillary street-level images, GeoChain pairs each image with a 21-step chain-of-thought (CoT) question sequence (over 30 million Q&A pairs). |
| Approach: | They propose a large-scale benchmark for evaluating step-by-step geographic reasoning in multimodal large language models (MLLMs) they pair each image with a 21-step chain-of-thought (CoT) question sequence (over 30 million Q&A pairs) |
| Outcome: | The proposed benchmark pairs 1.46 million images with a 21-step chain-of-thought (CoT) question sequence (over 30 million Q&A pairs) |
Distorted or Fabricated? A Survey on Hallucination in Video LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Despite significant advances in video-language modeling, hallucinations remain a persistent challenge in video large language models. |
| Approach: | They present a systematic taxonomy that categorizes hallucinations into two core types: dynamic distortion and content fabrication. |
| Outcome: | The proposed taxonomy categorizes hallucinations into two core types: dynamic distortion and content fabrication. |
CAVE : Detecting and Explaining Commonsense Anomalies in Visual Environments (2025.emnlp-main)
Copied to clipboard
Rishika Bhagwatkar, Syrielle Montariol, Angelika Romanou, Beatriz Borges, Irina Rish, Antoine Bosselut
| Challenge: | a new benchmark for computer vision fails to capture richness and unpredictability of real-world anomalies . state-of-the-art VLMs struggle with visual anomaly perception and commonsense reasoning . elucidating the nature of anomalies is a fundamental human trait . |
| Approach: | They propose a benchmark for visual anomalies that includes annotations for visual grounding and categorizing anomalies based on their visual manifestations, their complexity, severity, and commonness. |
| Outcome: | The proposed benchmark improves on existing vision models by incorporating visual annotations. |
Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for deciphering ancient Chinese Oracle Bone Script (OBS) treat deciphering as a closed-set image recognition problem, which fails to bridge the "interpretation gap" . |
| Approach: | They propose a vision-language model framework that integrates a VLM and an LLM to automate a reasoning chain of component identification and knowledge retrieval. |
| Outcome: | The proposed framework yields more detailed and precise decipherments compared to baseline methods. |
Thinking Like a Botanist: Challenging Multimodal Language Models with Intent Driven Chain-of-Inquiry (2026.findings-acl)
Copied to clipboard
Syed Nazmus Sakib, Nafiul Haque, Shahrear Bin Amin, Hasan Muhammad Abdullah, Md Mehedi Hasan, Mohammad Zabed Hossain, Shifat E. Arman
| Challenge: | Visual question-based reasoning is a key component of vision-language models. |
| Approach: | They propose a framework for visual question-answering that integrates visual intent with visual severity to improve diagnostic accuracy. |
| Outcome: | The proposed framework improves diagnostic correctness, reduces hallucination, and increases reasoning efficiency. |
Switching Heads and Softening Tokens: Turnkey Solutions to Visually Grounded Document QA (2026.findings-acl)
Copied to clipboard
| Challenge: | Document Question Answering lacks robust, end-to-end solutions capable of handling complex, multi-answer queries without reliance on ad-hoc processing. |
| Approach: | They propose a single-head architecture where coordinates are represented as special tokens within the unified vocabulary. |
| Outcome: | The proposed architectures improve visual grounding but lack spatial precision bound by discretization. |
Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models (2026.acl-long)
Copied to clipboard
Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Vatsal Sharan, Robin Jia, Willie Neiswanger
| Challenge: | Steering methods have emerged as effective tools for guiding large language models’ behavior, yet multimodal large language model lacks comparable techniques due to architectural diversity and limited availability of multimodal steering vectors. |
| Approach: | They validate steering vectors derived solely from text-only LLM backbones and use a cross-modal transfer technique to reuse existing interpretability tools. |
| Outcome: | The proposed steering vectors can guide and enhance multimodal models using SPAR, Mean Shift, and Linear Probing. |
The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form–Meaning Mapping (2026.acl-long)
Copied to clipboard
| Challenge: | a visual Iconicity test is used to evaluate vision–language models based on visual form and iconicity ratings. |
| Approach: | They propose a video-based benchmark to evaluate vision–language models on three tasks . they assess 17 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands . |
| Outcome: | The proposed benchmark evaluates 17 state-of-the-art VLMs on Sign Language of the Netherlands . they achieve moderate to strong alignment with human iconicity ratings, but fail to infer lexical meaning from visual form alone . |
KnowDR-REC: Auditing Knowledge-Conditioned Visual Grounding in Referring Expression Comprehension (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics suggest that Multimodal large language models have acquired fine-grained visual grounding capabilities. |
| Approach: | They propose a benchmark to assess Referring Expression Comprehension (REC) that uses intra-image visual cues to localize target objects and a controllable evaluation mechanism to test sensitivity to fine-grained factual changes. |
| Outcome: | The proposed benchmarks show that multimodal large language models have a high level of performance on the RefCOCO family of benchmarks. |
DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models may deviate from this pattern due to attention drift and underutilization of visual evidence. |
| Approach: | They propose a Dual-Indicator Guided Contrastive Alignment (DICA) that tracks visual attention and output image correlations to improve visual grounding. |
| Outcome: | The proposed model outperforms existing approaches and significantly reduces hallucinations. |
Multimodal Dual-Path Decoding for Medical Report Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Current methods for radiology report generation rely on encoder-decoder based frameworks that fail to integrate multimodal clinical evidence with domain-specific knowledge. |
| Approach: | They propose a multimodal dual-path framework that synergistically integrates large vision-language models and large language models for radiology report generation. |
| Outcome: | The proposed framework improves on the public MIMIC-CXR benchmark and shows that it is superior to state-of-the-art models. |
MMTabReal: Real-World Benchmark for Multimodal Table Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal tables are ubiquitous in real applications but are difficult to evaluate in multimodal large language models. |
| Approach: | They propose a multimodal table benchmark that compares 500 real-world tables with 4021 question–answer pairs. |
| Outcome: | MMtabReal spans four question types, five reasoning categories, and eight structural archetypes. |
VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing verbalized confidence calibration methods for large vision language models optimize a single holistic confidence score using binary answer-level correctness. |
| Approach: | They propose a reinforcement learning framework that explicitly decouples confidence into visual and reasoning confidence. |
| Outcome: | Experiments show that the proposed framework decouples confidence into visual and reasoning confidence while suppressing ungrounded hallucinations while preserving valid perception. |
FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting (2026.findings-acl)
Copied to clipboard
| Challenge: | FineState-Bench evaluates whether an agent can correctly ground an instruction to the intended UI control and reach the exact target state. |
| Approach: | They propose a benchmark that evaluates whether an agent can correctly ground an instruction to the intended UI control and reach the exact target state. |
| Outcome: | The proposed benchmark evaluates whether an agent can ground an instruction to the intended UI control and reach the exact target state. |